Image processing device, image processing method, and non-transitory recording medium

ABSTRACT

An image processing device includes a memory storing a program, and at least one processor that executes the program. The processor sets an exposure condition for video imaging, acquires an image of the video imaged under the exposure condition, and calculates a brightness distribution from the image. The exposure condition includes a first exposure condition for causing the imaging device to capture an image using a first exposure value, and a second exposure condition causing the imaging device to capture an image using an exposure value higher than the first exposure condition and/or causing the imaging device to capture an image using a lower exposure value than the first exposure condition. The processor sets the first exposure condition or sets the first and the second exposure condition, based on the brightness distribution of the image captured under the first exposure condition after the first exposure condition is set.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Patent Application No.2019-173594, filed on Sep. 24, 2019, and Japanese Patent Application No.2020-053725, filed on Mar. 25, 2020, the entire disclosure of which isincorporated by reference herein.

FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a non-transitory recording medium.

BACKGROUND

There are techniques for detecting the face of a person from an imagecaptured by an imaging device and performing facial recognition. PatentLiterature 1 discloses a device that provides images suitable for facedetection and facial recognition, even when there arebrightness/darkness differences in the imaging area.

In the invention described in Japanese Unexamined Patent ApplicationPublication No. 2007-251558, the exposure value of the imaging device iscontrolled so that the brightness distribution of an image, of a regionof a detected face, is the same as the brightness distribution of areference image serving as a basis for facial recognition.

SUMMARY

An image processing device of the present disclosure includes:

a memory storing a program; and

at least one processor executing the program stored in the memory,wherein the processor executing the program sets an exposure conditionfor imaging a monitoring video, acquires an image included in themonitoring video imaged under the set exposure condition, and calculatesa brightness distribution from the acquired image, the exposurecondition includes (1) a first exposure condition for causing an imagingdevice to capture the image using a first exposure value, and (2) asecond exposure condition for at least one of causing the imaging deviceto capture the image using an exposure value higher than the firstexposure value or causing the imaging device to capture the image usingan exposure value lower than the first exposure value, and the processorsets the first exposure condition or set the first exposure conditionand the second exposure condition as the exposure condition for imagingbased on the brightness distribution of the image captured under thefirst exposure condition after the first exposure condition is set.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained whenthe following detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 is a drawing illustrating a facial recognition system accordingto Embodiment 1 of the present disclosure;

FIG. 2 is a drawing illustrating an overview of the processing flow ofthe facial recognition system according to Embodiment 1 of the presentdisclosure;

FIG. 3 is a block diagram of an image processing device according toEmbodiment 1 of the present disclosure;

FIG. 4A is a drawing illustrating a brightness distribution of an imagecaptured by the image processing device according to Embodiment 1 of thepresent disclosure, and illustrates the brightness distribution whenimaging using a typical exposure value;

FIG. 4B is a drawing of a brightness distribution when imaging using atypical exposure value and a positive exposure value;

FIG. 4C is a drawing of a brightness distribution when imaging using atypical exposure value, a positive exposure value, and a negativeexposure value;

FIG. 5A illustrates the switching of the frames of the image processingdevice according to Embodiment 1 of the present disclosure, and is adrawing of a case of imaging under typical settings;

FIG. 5B is a drawing of a case of alternately imaging under exposuresettings of typical and positive exposure values;

FIG. 5C is a drawing of a case of alternately imaging under exposuresettings of a typical exposure value, a positive exposure value, and anegative exposure value;

FIG. 6 is a flowchart of exposure correction processing 1 of the imageprocessing device according to Embodiment 1 of the present disclosure;

FIG. 7 is a flowchart of exposure tracking processing of the imageprocessing device according to Embodiment 1 of the present disclosure;

FIG. 8 is a flowchart of image synthesis processing 1 of the imageprocessing device according to Embodiment 1 of the present disclosure;

FIG. 9A is an image captured under typical settings;

FIG. 9B is an image captured using a positive exposure value or anegative exposure value;

FIG. 9C is a drawing illustrating an image output by the image synthesisprocessing 1;

FIG. 10 is a flowchart of image synthesis processing 2 of the imageprocessing device according to Embodiment 1 of the present disclosure;

FIG. 11 is a drawing illustrating a comparison, performed in FIG. 10, ofdistances between a center of a brightness distribution of a face and acenter of brightness levels;

FIG. 12 is a drawing illustrating an image output by the image synthesisprocessing 2;

FIG. 13 is a drawing illustrating an example in which an image obtainedfrom the image processing device according to Embodiment 1 of thepresent disclosure is converted from 12-bit to 8-bit;

FIG. 14 is a drawing illustrating an example of a gamma curve of gammacorrection of the image processing device according to Embodiment 1 ofthe present disclosure;

FIG. 15A illustrates switching of frames of an image processing deviceaccording to Embodiment 2 of the present disclosure, and illustrates acase of imaging under settings for switching frames every single frame;

FIG. 15B is a drawing of case of imaging four frames under a firstexposure condition and, then, imaging one frame under a second exposurecondition;

FIG. 16 is a drawing illustrating a synthesis method of an image of theimage processing device according to Embodiment 2 of the presentdisclosure; and

FIG. 17 is a flowchart of exposure correction processing 2 of the imageprocessing device according to Embodiment 2 of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, an image processing device according to embodiments of thepresent disclosure is described in detail while referencing thedrawings.

In one example, the image processing device according to the embodimentsof the present disclosure generates image data for causing a facialrecognition device of a facial recognition system to perform facialrecognition at a railway train station in order to mine data related tothe working start time and the working end time of railway users. Thisimage data can also be used in office and event security, and the like.Note that the number of persons captured in an image is not particularlylimited provided that the face images are not excessively small but, inthe following description, for the sake of ease of description, thenumber of persons captured in an image is set to two.

Embodiment 1 Configuration of Facial Recognition System

As illustrated in FIG. 1, a facial recognition system 1 includes animage processing device 10 and a facial recognition device 80. The imageprocessing device 10 captures an image of a person 100 (101, 102), whichis a recognition target area, present in an imaging range L of thefacial recognition system 1, performs exposure correction processing 1and image processing (described later), and sends image data suitablefor facial recognition to the facial recognition device 80. A highbrightness area 200, a low brightness area 300, and the like areproduced in the imaging range L. The high brightness area 200 is an areain which sunlight enters from outside through a window, for example. Thelow brightness area 300 is an area in which shadows are produced bypillars or trains arriving at and departing from the train station. Insome cases, facial recognition cannot be correctly performed on thepersons 101, 102 that are in the high brightness area 200, the lowbrightness area 300, or the like in which the brightness/darknessdifference is great. As such, the image processing device 10 performsexposure correction processing 1, image processing, and the like andprovides image data suitable for facial recognition so that the facialrecognition device 80 can perform facial recognition of the person 101that is in the high brightness area 200 and of the person 102 that is inthe low brightness area 300.

An overview of the processing performed by the facial recognition system1 is described while referencing FIG. 2. The image captured by theimaging device is a 12-bit Bayer image, and this image is developed andtone corrected to generate a YUV image compressed to 8 bits. Facedetection of the generated image is performed in the image processingdevice 10, and face matching is performed in the facial recognitiondevice 80.

Configuration of Image Processing Device

As illustrated in FIG. 3, the image processing device 10 includes acontroller 20, a storage 30, an imager 40, a communicator 50, a display60, and an inputter 70.

The controller 20 includes a central processing unit (CPU) or the like.The controller 20 executes a program or the like stored in the storage30 to realize the functions of the hereinafter described components (anexposure setter 21, an image acquirer 22, a brightness distributioncalculator 23, a determiner 24, a corrector 25, an image processor 26,an image sender 27, and an operator 28). Additionally, the controller 20includes a clock (not illustrated in the drawings), and is capable ofacquiring the current time and date, counting elapsed time, and thelike.

The storage 30 includes a read-only memory (ROM), a random-access memory(RAM), or the like. The ROM is configured in part or in whole fromelectrically rewritable memory (flash memory or the like). The storage30 functionally includes an exposure condition storage section 31, animage storage section 32, and a brightness distribution storage section33. Programs to be executed by the CPU of the controller 20 and dataneeded in advance to execute these programs are stored in the ROM. Datathat is created or modified during the execution of the programs isstored in the RAM.

An exposure condition set by the exposure setter 21 (described later) isstored in the exposure condition storage section 31. The exposurecondition is set by the exposure setter 21, but can be corrected by thecorrector 25, and can be changed and added by a user. Note that,typically, a first condition is set for the exposure condition and, atan initial setting, an exposure value EV0 of a typical state (exposurevalue at which a portion of 18% gray reflectance of thebrightness/darkness of the subject is correctly imaged) is set.

An image V acquired by the image acquirer 22 is stored in the imagestorage section 32. Additionally, an image V1 processed by the imageprocessor 26 is stored in the image storage section 32.

A brightness distribution B calculated by the brightness distributioncalculator 23 is stored in the brightness distribution storage section33.

The imager 40 includes an imaging device 41 and a driving device 42.

In the present embodiment, the imaging device 41 includes aComplementary Metal Oxide Semiconductor (CMOS) camera. The imagingdevice 41 captures the image V under the exposure condition set by theexposure setter 21. The image V is a Bayer image, and is output in12-bit resolution.

The driving device 42 moves the position of the imaging device 41 inaccordance with commands from the operator 28 (described later) toadjust the imaging range L.

The communicator 50 includes a communication device 51 that is a modulethat enables communication with the facial recognition device 80,external devices, and the like. The communication device 51 is awireless module that, in cases in which communication with externaldevices is to be performed, includes an antenna. In one example, thecommunication device 51 is a wireless module for performing short-rangewireless communication based on Bluetooth (registered trademark). Byusing the communicator 50, the image processing device 10 can exchangeimage data and the like with the facial recognition device 80 and theexternal devices.

The display 60 includes a display device 61 includes a liquid crystaldisplay (LCD).

A thin film transistor (TFT), liquid crystal, organic EL, or the likecan be used as the display device 61. The images V, V1, the brightnessdistribution B of the image V calculated by the brightness distributioncalculator 23, and the like are displayed on the display device 61.

The inputter 70 is a resistive film touch panel that is provided inclose proximity to the display 60 or integrated with the display 60. Thetouch panel may be an infrared operation-type or projectedcapacitive-type touch panel. Alternatively, the inputter 70 may be akeyboard, a mouse, or the like instead of a touch panel. The user cancorrect the brightness distribution B displayed on the display 60 byperforming manual operations via the inputter 70.

Next, the functional configuration of the controller 20 of the imageprocessing device 10 is described. The controller 20 realizes thefunctions of the exposure setter 21, the image acquirer 22, thebrightness distribution calculator 23, the determiner 24, the corrector25, the image processor 26, the image sender 27, and the operator 28,and performs exposure correction, exposure tracking, image synthesis,and the like.

The exposure setter 21 sets the exposure condition for when the imager40 captures a monitoring video. The exposure condition is set based on acalculation result, calculated by the brightness distribution calculator23 (described later), of a brightness distribution D of the image V. Theset exposure condition is stored in the exposure condition storagesection 31. Additionally, the exposure setter 21 sets a second conditionwhen a peak P of the brightness distribution D calculated from theacquired image V is not in a predetermined range R (described later).

The image acquirer 22 causes the imager 40 to capture the monitoringvideo under the exposure condition set by the exposure setter 21, andacquires the image V included in the captured monitoring video. Theimage acquirer 22 stores the acquired image V in the image storagesection 32.

The brightness distribution calculator 23 calculates the brightnessdistribution D (distribution of numerical values obtained by assigningbrightnesses of 0% to 100% to 0 to 255) from the image V acquired by theimage acquirer 22. The brightness distribution calculator 23 stores thecalculated brightness distribution D of the image V in the brightnessdistribution storage section 33.

The determiner 24 determines if the peak P (P1, P2 . . . ) of thebrightness distribution D calculated by the brightness distributioncalculator 23 is in the predetermined range R, is less than thepredetermined range R, or exceeds the predetermined range R. Asillustrated in the brightness distribution diagrams of FIGS. 4A to 4C,in the present embodiment, the predetermined range R is a range of 63 to191, which is the two middle regions (MID_1 and MID_2) of four regionsobtained by dividing the brightness levels of 0 to 255 into fourregions. If the peak P is in the MID_1 and MID_2 regions, the influencesof whiteout and blackout on the image V are small even without adding aframe.

The corrector 25 performs correction so as to move the peak P to thecenter of the predetermined range R when the peak P of the brightnessdistribution D calculated by the brightness distribution calculator 23is in the predetermined range R. The correction method is describedlater.

The image processor 26 processes the image V stored in the image storagesection 32. A face image F2 in an image V2 of one exposure setting iscombined with an image V1 of another exposure setting to generate imagesVA, VB on which the facial recognition device 80 can perform facialrecognition.

The image sender 27 sends the acquired image V, the processed image V1,and the like to the facial recognition device 80.

The operator 28 sends, to the driving device 42, a command for movingthe imaging range L of the imager 40.

The functional configuration of the controller 20 is described above.Next, details of the exposure correction performed by the imageprocessing device 10 are described using, as examples, cases in whichthe brightness distribution acquired from the captured image is asillustrated in FIGS. 4A to 4C.

First, when the peak P of the brightness distribution D is in thepredetermined range R (brightness level of 63 to 191) as illustrated inFIG. 4A, the corrector 25 calculates

Δ=127/P  (1)

based on the peak P of the brightness distribution D. The corrector 25multiplies A by the peak P, fine-adjusts the first condition so that thepeak P is positioned at the center of the predetermined range R, andcaptures the image V1. In this case, as illustrated in FIG. 5A, onlyimaging of a frame of a reference exposure Ref, which is the firstexposure condition, is performed, without an additional frame.

Next, when, as illustrated in FIG. 4B, the peak P1 of the brightnessdistribution D is less than the predetermined range R and the peak P2 isin the predetermined range R, the corrector 25 multiples the peak P1 bythe Δ obtained in Equation (1) from the peak P1 of the brightnessdistribution D, and sets the second condition that has an increasedexposure value so as to position the peak P1 at the center of apredetermined range RE As illustrated in FIG. 5B, the imaging device 41performs imaging by alternately switching frames based on the exposurevalue of the first exposure condition (reference exposure Ref) and theexposure value of the second exposure condition (Positive exposure).Note that, when the peak P1 that is outside the predetermined range Rexceeds the predetermined range R instead of being less than thepredetermined range R, the second exposure condition is obtained bydecreasing the exposure value using Equation (1). In this case, thesecond exposure condition is Negative exposure, and the Positiveexposure frame of FIG. 5B is replaced with a Negative exposure frame.That is, frames of the reference exposure Ref and the Negative exposureare alternately imaged.

When, as illustrated in FIG. 4C, the peak P1 of the brightnessdistribution D is less than the predetermined range R and the peak P2exceeds the predetermined range R, the corrector 25 multiples the peakP1 by the Δ obtained in Equation (1) from the peak P1 of the brightnessdistribution D, and sets a second exposure condition that has anincreased exposure value so as to position the peak P1 at the center ofthe predetermined range R1 and, also, multiples the peak P2 by the Δobtained in Equation (1) from the peak P2 of the brightness distributionD, and sets a second exposure condition that has a decreased exposurevalue so as to position the peak P2 at the center of the predeterminedrange R1. As illustrated in FIG. 5C, the imaging device 41 performsimaging by sequentially switching between three frames based on theexposure value of the first exposure condition (the reference exposureRef) and the exposure values of the two second exposure conditions (thePositive exposure and the Negative exposure).

Facial Recognition Device

In one example, the facial recognition device 80 is a device in whicheigenfaces generated by principle component analysis are used as afacial recognition algorithm. The facial recognition device 80 performsfacial recognition (two-dimensional facial recognition) using the imagedata sent from the image processing device 10.

Various processing performed by image processing device Next, theexposure correction processing 1, the exposure tracking processing, andthe image synthesis processings 1 and 2 performed by the imageprocessing device 10 are described using flowcharts. Note that theexposure correction processing 1, the exposure tracking processing, andthe image synthesis processings 1 and 2 can be executed in parallel.

Exposure Correction Processing 1

The flow of the exposure correction processing 1 performed by the imageprocessing device 10 is described while referencing FIG. 6. Due to theexposure correction processing 1, images can be captured by switchingthe exposure settings of the frames in accordance with the detectionsituation of a face in the image, as illustrated in FIGS. 5A to 5C. As aresult, substantial reductions in the frame rate (perform imaging usingonly the reference frame when an additional frame is unnecessary) can besuppressed, and an additional frame can be used as necessary for thepersons 101, 102 in the high brightness area 200 and the low brightnessarea 300 illustrated in FIG. 1 where the brightness/darkness differenceis great. As such, the facial recognition device 80 can perform facialrecognition more accurately.

First, the exposure setter 21 sets an exposure value EV (Exposure Value)0 (aperture value F1, exposure time one second) of the first condition(reference frame), and stores the first exposure condition in theexposure condition storage section 31 (step S1).

Next, the image acquirer 22 references the exposure condition storagesection 31 and causes the imager 40 to capture the image V in theimaging range L under the first exposure condition stored in theexposure condition storage section 31. In cases of returning from stepS12, when an additional frame has been set in step S7 and/or step S9, asillustrated in FIG. 5B or FIG. 5C, the imager 40 is caused toalternately or sequentially image the reference frame and one or twoadditional frames under the first exposure condition and the secondexposure condition described above (step S2).

The imaging device 41 of the imager 40 captures the image V in theimaging range L under the first exposure condition, and the imageacquirer 22 acquires the captured image V. The image acquirer 22 storesthe acquired image V in the image storage section 32. As described inthe previous paragraph, when an additional frame is set, imaging isperformed for the additional frame under the second exposure condition(described later) and the acquired image is stored in a similar manner(step S3).

The brightness distribution calculator 23 references the image storagesection 32 and calculates the brightness distribution D of the image Vstored in the image storage section 32. The brightness distributioncalculator 23 stores the calculated brightness distribution D of theimage V in the brightness distribution storage section 33 (step S4).

The brightness distribution calculator 23 calculates the peak P (P1, P2. . . ) of the brightness distribution D (step S5).

The determiner 24 determines whether there is a peak P that is less thanbrightness level 63 (step S6). When there is a peak P that is less thanbrightness level 63 (step S6; Yes), an additional frame of the exposurevalue of the second condition, which has an increased exposure value, isset. The exposure setter 21 calculates, from the brightness level of thepeak P of the brightness distribution D, a Δ that satisfies Equation(1), sets a value, obtained by multiplying the A by brightness level 127of the reference exposure, as the exposure value of the second exposurecondition, and stores the exposure value of the second exposurecondition in the exposure condition storage section 31 (step S7). Then,step S8 is executed.

When there is not a peak P that is less than brightness level 63 (stepS6; No), step S8 is executed.

The determiner 24 determines whether there is a peak P that exceedsbrightness level 191 (step S8). When there is a peak P that exceedsbrightness level 191 (step S8; Yes), an additional frame of the exposurevalue of the second exposure condition, which has a decreased exposurevalue, is set. The exposure setter 21 calculates the Δ of Equation (1)from the peak P of the brightness distribution D, sets a value, obtainedby multiplying the Δ by brightness level 127 of the reference exposure,as the exposure value of the second exposure condition, and stores theexposure value of the second exposure condition in the exposurecondition storage section 31 (step S9). Then, step S10 is executed.

When there is not a peak P that exceeds brightness level 191 (step S8;No), step S10 is executed.

The determiner 24 determines whether there is an additional frame (stepS10). When there is an additional frame (step S10; Yes), step S12 isexecuted. When there is not an additional frame (step S10; No), step S11is executed.

In step S11, the corrector 25 corrects the first exposure condition bymultiplying the Δ, calculated by Equation (1), by the exposure value setin step S1 so as to achieve the brightness level 127, which is themedian value of the brightness values, in the first exposure condition.Then, step S12 is executed.

When ending (step S12; Yes), the processing is ended. When not ending(step S12; No), step S2 is executed.

Thus, due to the exposure correction processing 1, in the brightnessdistribution D calculated by the brightness distribution calculator 23,when there is a peak P of the brightness distribution D of the capturedimage outside the predetermined range R, the exposure setter 21 sets anadditional frame of an exposure value that corresponds to the brightnesslevel of the peak P, and the image acquirer 22 acquires images capturedin the reference exposure and the additional exposure frames. As aresult, suitable image data can be sent to the facial recognition device80. Additionally, the reference exposure frame is set even when theadditional frame is set and, as such, the occurrence of face imagedetection omissions can be suppressed.

Exposure Tracking Processing

The exposure tracking processing is described using FIG. 7. Thisprocessing is for controlling the fluctuation speed of exposure for eachframe. In the case of a video to be viewed by humans, frame continuityis important. As such, restrictions are imposed on the fluctuation rangeof exposure for each frame with respect to exposure change. Meanwhile,in videos for machine authentication, changing to the optimal exposureas quickly as possible is more important than frame continuity. The goalis to acquire, as quickly as possible, images suitable for facialrecognition. Note that exposure tracking is performed when facedetection is carried out and, as such, the exposure tracking processingis executed when face detection is carried out in the reference exposureframe and/or the additional frame.

The determiner 24 determines whether a face is detected in the referenceexposure frame, in which imaging under the first condition is performed(step S30). When a face is detected in the reference exposure frame(step S30; Yes), the exposure values are tracked at low speed toprioritize facial recognition of the current state (step S31), and theexposure tracking processing is ended.

When a face is not detected in the reference exposure frame, that is,when a face is detected in the additional frame (step S30; No), theexposure values are tracked at high speed (step S32), and the exposuretracking processing is ended.

Thus, due to this exposure tracking processing, the exposure values canbe tracked as quickly as possible while reducing undesirable effectssuch as getting lost in following processing of a person for whom facedetection is already performed in the reference frame.

Image Synthesis Processing 1

Next, the image synthesis processing 1 is described using FIGS. 8 and 9.The image synthesis processing 1 is processing for synthesizing theimage to be sent to the facial recognition device 80 in a case in whichthe image V is captured in a plurality of frames imaged under thereference and additional exposure conditions in the exposure correctionprocessing 1 (hereinafter, the “plurality of frames imaged under thereference and additional exposure conditions” is simply referred to as“plurality of frames”; notation in the drawings is also made as“plurality of frames”).

First, in FIG. 8, the determiner 24 determines whether face detection iscarried out in the captured image V (step S40). If face detection iscarried out in the image V (step S40: Yes), step S41 is executed.

The determiner 24 determines whether the imaging device 41 captures theimage V in a plurality of frames (step S41). When the imaging device 41captures the image V in a plurality of frames (step S41; Yes), step S42is executed.

In step S42, the determiner 24 determines whether there is a face imagethat is not detected in the reference exposure frame and is detectedonly in the additional exposure frame. To accomplish this, thedeterminer 24 determines whether each of the face images detected in theadditional exposure frame matches a face image detected in the referenceexposure frame. Methods for determining whether the face images match inboth frames include calculating feature values of the regions of theface images in both frames and performing matching using a predeterminedalgorithm; and calculating the difference between the feature values ofthe face images in both frames and determining that the face images arethe same if the calculated difference is within a certain range. Theface image, among the face images detected in the additional exposureframe, that does not match a face image detected in the referenceexposure frame is the face image obtained in step S42. Note that, incases in which the imaging target does not move or barely moves, it maybe determined whether or not the face images match based on thepositions of the face images or the like.

When there is not a face image that is not detected in the referenceexposure frame and is only detected in the additional exposure frame(step S42; No), the image of the reference exposure frame is selectedand sent to the facial recognition device 80 (step S43), and the imagesynthesis processing 1 is ended. When there is a face image that is notdetected in the reference exposure frame and is only detected in theadditional exposure frame (step S42; Yes), the reference exposure frameis selected, the image processor 26 trims the face image F2 detectedonly in the additional exposure frame, and synthesizes the image V1 bycombining the trimmed face image F2 with the reference exposure frame.Then, the image sender 27 sends the image V1 to the facial recognitiondevice 80 (step S44), and the image synthesis processing 1 is ended. Inan example using FIG. 9, only the face image F2A, which is not detectedin the image V1 of the reference exposure frame illustrated in FIG. 9Aand is detected only in the image V2 of the additional exposure frameillustrated in FIG. 9B, is trimmed and combined with the image V1 toobtain an image VA illustrated in FIG. 9C, and this image VA is sent tothe facial recognition device 80.

When face detection is not carried out in the captured image (step S40;No), and when the imaging device 41 does not capture the image V in aplurality of frames, that is, when the image V is captured only in thereference exposure frame (step S41; No), the image sender 27 sends theimage V of the reference exposure frame to the facial recognition device80 without modification (step S43), and the image synthesis processing 1is ended.

Due to the image synthesis processing 1, image processing is executed inwhich only the face image (target) that cannot be detected in thereference exposure frame and can only be detected in the additionalexposure frame is combined with the reference exposure frame. As aresult, the image sent to the facial recognition device 80 is an imageof the reference exposure frame, and only the face image that cannot bedetected in the reference exposure frame is combined with the image ofthe reference exposure frame. As such, the amount of image data that issent can be reduced.

Image Synthesis Processing 2

Next, the image synthesis processing 2 is described using FIG. 10. Inthe image synthesis processing 1, the face image of the referenceexposure frame is selected when the face image is detected in both thereference exposure frame and the additional exposure frame. However, inthe image synthesis processing 2, when the face image is detected inboth the reference exposure frame and the additional exposure frame, theface image in which the face image is expressed better is selected (forexample, in FIG. 11, the face image of the additional exposure frame, inwhich the center A of the brightness distribution of the face is closerto the center C of the 12-bit brightness levels than the center B of thebrightness distribution of the face of the reference exposure frame, isselected). By selecting the face image in which the center of thebrightness distribution of the face is close to the center C of the12-bit brightness levels, a wider brightness distribution of the facecan be reflected in the image to be sent. The image synthesis processing2 differs from the image synthesis processing 1 in that, for example, ina case in which the image V1 illustrated in FIG. 9A is captured in thereference exposure frame and the image V2 illustrated in FIG. 9B iscaptured in the additional exposure frame, it is determined whether touse the face image F1A illustrated in FIG. 9A or the face image F2Billustrated in FIG. 9B for the face image FB of the synthesized image VBillustrated in FIG. 12.

The processing of steps S40 and S41 are the same as in the imagesynthesis processing 1. The determiner 24 determines whether facedetection is carried out in the captured image V (step S40). When facedetection is carried out in the image V (step S40; Yes), the determiner24 determines whether the imaging device 41 captures the image V in aplurality of frames (step S41). When the imaging device 41 captures theimage V in a plurality of frames (step S41; Yes), step S52 is executed.When face detection is not carried out in the image V (step S40; No),and the imaging device 41 does not capture the image V in a plurality offrames (step S41; No), the image synthesis processing 2 is ended.

In step S52, the determiner 24 determines whether there is a face imagethat is detected in both the reference exposure frame and the additionalexposure frame. To accomplish this, as in step S42, the determiner 24determines whether each face image detected in the additional exposureframe matches a face image detected in the reference exposure frame. Themethod for determining whether the face images in both frames match isthe same as the method described in step S42. When there is a face imagedetected in both the reference exposure frame and the additionalexposure frame (step S52; Yes), step S53 is executed. When there is nota face image detected in both the reference exposure frame and theadditional exposure frame (step S52; No), step S55 is executed.

In step S53, the determiner 24 determines whether the center of thebrightness distribution of the face image detected in both the referenceexposure frame and the additional exposure frame is closer to the centerof the brightness levels in the additional exposure frame than to thecenter of the brightness levels in the reference exposure frame (stepS53). That is, the determiner 24 determines whether a state such asillustrated in FIG. 11 is realized. This is processing for selecting theface image of the frame in which the center of the brightnessdistribution of the detected face image is close to the center of thebrightness levels in a case of 12-bit resolution.

When the center of the brightness distribution of the face imagedetected in both the reference exposure frame and the additionalexposure frame is closer to the center C of the brightness levels of theadditional exposure frame in a case of 12-bit resolution than to thecenter C of the brightness levels of the reference exposure frame in acase of 12-bit resolution (step S53; Yes), that face image is trimmedfrom the additional exposure frame, combined with the image of thereference exposure frame (step S54), and step S55 is executed. When thecenter of the brightness distribution of the face image detected in boththe reference exposure frame and the additional exposure frame is notcloser to the center of the brightness levels of the additional exposureframe than to the center of the brightness levels of the referenceexposure frame (including cases in which the distances are equal) (stepS53; No), step S55 is executed without performing the image synthesis.

In step S55, the determiner 24 determines whether there is a face imagethat is not detected in the reference exposure frame yet is detected inthe additional exposure frame only. To accomplish this, as in steps S42and S52, the determiner 24 determines whether each face image detectedin the additional exposure frame matches a face image detected in thereference exposure frame. The method for determining whether the faceimages in both frames match is the same as the method described in stepS42.

In step S55, when there is a face image that is not detected in thereference exposure frame and is detected only in the additional exposureframe (step S55; Yes), the face image detected only in the additionalexposure frame is trimmed and combined with the image captured inreference exposure frame (in a case of step S53; Yes, the image withwhich the face image detected in the additional exposure frame isalready combined in part), the synthesized image is sent to the facialrecognition device 80 (step S57), and the image synthesis processing 2is ended.

When there is not a face image that is not detected in the referenceexposure frame and is detected only in the additional exposure frame(step S55; No), the image sender 27 sends the image captured in thereference exposure frame without modification (in a case of step S53;Yes, the image with which the face image detected in the additionalexposure frame is already combined in part) to the facial recognitiondevice 80, and the image synthesis processing 2 is ended.

Thus, due to the image synthesis processing 2, when a face image(target) is detected in both the reference exposure frame and theadditional exposure frame, the face image of the frame in which thecenter of the brightness distribution of the face image is close to thecenter of the brightness levels in a case of 12-bit resolution isselected. Improvements in facial recognition accuracy are anticipated byselecting the face image in which the face image is expressed better.

Embodiment 2

In Embodiment 1, when there is a peak P of the brightness distribution Dof the captured image outside the predetermined range R, alternatelyimaging the reference exposure frame and the additional exposure frameone frame at a time is performed. However, in the present embodiment,when imaging is performed using the reference exposure frame and theadditional exposure frame, the frequency of the switching of the framesis adjusted.

In the present embodiment, the configuration of the facial recognitionsystem 1 and the configuration of the image processing device 10 are thesame as in Embodiment 1. With the exception of the exposure correctionprocessing 1, the various processings performed in Embodiment 2 areperformed in the same manner as in Embodiment 1. However, some functionsare added to the controller 20 and the storage 30. Hereinafter, theseadditional functions are described.

In cases in which the second exposure condition is set, the exposuresetter 21 sets a frequency of switching between imaging under the firstexposure condition and imaging under the second exposure condition. Whena face image that is the primary subject is detected under only thesecond exposure condition, as illustrated in FIG. 15A, imaging under thefirst exposure condition and imaging under the second exposure conditionare alternately switched one frame at a time. When the same face imageis detected under the first exposure condition and the second exposurecondition, as illustrated in FIG. 15B, switching is carried out suchthat four frames are imaged under the first exposure condition and thenone frame is imaged under the second exposure condition. The exposuresetter 21 also stores the frequency at which the imaging is switched inthe exposure condition storage section 31. Note that the determiner 24determines the exposure condition under which the face image isdetected.

The determiner 24 determines whether the face image can be confirmed inthe image captured under the first exposure condition and in the imagecaptured under the second exposure condition. The determination resultis sent to the exposure setter 21.

In the present embodiment, an exposure value of +2EV is set in theexposure condition storage section 31 as the second exposure condition.

The additional functions are described above. Hereinafter, a specificdescription is given, while referencing FIGS. 4B, 15A, 15B, and 16, ofexposure correction performed by the image processing device 10 of thepresent embodiment.

As illustrated in FIG. 4B, when the peak P1 of the brightnessdistribution D is less than the predetermined range R and the peak P2 isin the predetermined range R, as illustrated in FIG. 15B, the imagingdevice 41 switches the frames based on the exposure values of the firstexposure condition (reference exposure Ref (0EV)) and the secondexposure condition (Positive exposure (+2EV)), and performs imaging at30 fps. At this time, when a face image that is the primary subject isdetected only under the second exposure condition, as illustrated inFIG. 15A, imaging under the first exposure condition and imaging underthe second exposure condition are alternately switched one frame at atime. The frame rate is reduced to ½ because the face image is detectedonly under the second exposure condition. Since an image that iscaptured under the first exposure condition is selected and used for thebackground, the frame rate of the background is reduced to ½, or 15 fps.

When the same face image is detected under the first exposure conditionand the second exposure condition, or when the face image is notdetected under the second exposure condition, as illustrated in FIG.15B, the frames are switched such that four frames are imaged under thefirst exposure condition and then one frame is imaged under the secondexposure condition, and imaging is performed at 30 fps. Since four outof five frames are imaged under the first exposure condition, the framerate of the reference frame is reduced to ⅘. Since an image that iscaptured under the first exposure condition is selected and used as thebackground, the frame rate of the background is reduced to ⅘, or 24 fps.When performing imaging under the first exposure condition and thesecond exposure condition in Embodiment 1, reductions of the frame rateof the reference frame can be made smaller than in the case illustratedin FIG. 15A.

In the cases illustrated in FIGS. 15A and 15B as well, as illustrated inFIG. 16, placement coordinates for the face image, captured under thesecond exposure condition, are made to correspond with and are combinedwith the face image and the background captured under the first exposurecondition, a MAP image is generated, and the image when imaging undereach exposure condition is updated.

Exposure Correction Processing 2

Next, exposure correction processing 2 performed in the presentembodiment instead of the exposure correction processing 1 performed inEmbodiment 1 is described while referencing FIG. 17. Due to the exposurecorrection processing 2, the exposure settings of the frames can beswitched in accordance with the detection situation of faces in theimages, and the switching frequency of the frames can be adjusted tocapture images. Note that, in the exposure correction processing 2, forthe sake of ease of description, the second exposure condition is only+2EV, which is the value that is set when there is a peak P that is lessthan brightness level 63.

In steps S1 to S5 of the exposure correction processing 2, the sameprocessing as in the exposure correction processing 1 is performed and,then, step S60 is executed.

In step S60, the determiner 24 determines whether there is, in thebrightness distribution D of the image V in the imaging range L, a peakP that is less than brightness level 63 such as illustrated in FIGS. 4Band 4C, for example.

When there is not a peak P that is less than brightness level 63 (stepS60; No), step S61 is executed.

When there is a peak P that is less than brightness level 63 (step S60;Yes), the exposure setter 21 sets an additional frame of +2EV, which isthe exposure value of the second condition that has an increasedexposure value. The determiner 24 determines whether the face isdetected only in the frame imaged under the second exposure condition.

When the face is detected only in the frame imaged under the secondexposure condition (step S62; Yes), the determiner 24 sends thedetermination result to the exposure setter 21. The exposure setter 21sets so that imaging under the first exposure condition and imagingunder the second exposure condition are alternately switched one frameat a time (step S63). After the setting, step S61 is executed.

When the face is not detected in the frame imaged under the secondexposure condition or the face is detected in the frames imaged underthe first and second exposure conditions (step S62; No), the determiner24 sends the determination result to the exposure setter 21. Theexposure setter 21 sets so that switching is carried out such that fourframes are imaged under the first exposure condition and then one frameis imaged under the second exposure condition (step S64). After thesetting, step S61 is executed.

When ending (step S61; Yes), the exposure correction processing 2 isended. When not ending (step S61; No), step S2 is executed.

Thus, due to the exposure correction processing 2, in the brightnessdistribution D calculated by the brightness distribution calculator 23,when there is a peak P of the brightness distribution D of the capturedimage that is less than the predetermined range R (less than brightnesslevel 63), the exposure setter 21 sets an additional frame of theexposure value +2EV and sets the switching frequency of the framesaccording to whether the face is detected only in the additionalexposure frame, and the image acquirer 22 acquires images captured inthe reference exposure and the additional exposure frames. When the faceis detected only in the additional exposure frame, the referenceexposure and the additional exposure frames are alternately imaged oneframe at a time; and when the same face is detected in the referenceexposure and the additional exposure frames, or when the face image isnot detected in the additional exposure frame, the frequency of imagingunder the additional exposure is reduced (or the frequency of imagingunder the reference exposure is increased). As a result, image data thatis more suitable can be sent to the facial recognition device 80.Moreover, the reference exposure frame is set even when the additionalframe is set and, as such, the occurrence of face image detectionomissions can be suppressed.

Modified Examples

In the embodiments described above, the image processing device 10simply compresses the 12-bit Bayer image to 8 bits. However, aconfiguration can be made in which, for example, compression isperformed such that the tone of the face is maximized, as illustrated inFIG. 13. It is preferable that, after the brightness distributioncalculator 23 calculates the brightness distribution, a gamma table isused that aligns the center C of the brightness distribution of the facewith the center of the 8-bit brightness distribution and linearlyconverts from a point D to a point E, where reserved regions for acertain measurement error from a minimum value A and a maximum value Bof the brightness distribution of the face are secured. Additionally,gamma curve correction as illustrated in FIG. 14 may be carried out.Moreover, in FIG. 2, the gamma correction may have a two-stageconfiguration.

In the embodiments described above, the image processing device 10performs image processing and the like for the facial recognition device80. However, a configuration can be made in which the image processingdevice 10 performs image processing and the like for an imagerecognition device that performs person detection or objects (vehiclesor the like) detection in which, instead of a face, a person, an object,or the like is the target. Additionally, the target and the subject maybe the same or, for example, the target may be a face, and the subjectmay be a person.

In Embodiment 1 described above, as illustrated in FIGS. 5B and 5C theimage processing device 10 images by alternately changing frames eachframe. However, configurations can be made in which imaging is performedby alternately changing frames every two frames, three frames, or fourframes, or every one second, five seconds, or ten seconds.

In the embodiments described above, the image processing device 10 sendsa synthesized image or an image at one frame rate to the facialrecognition device 80. However, a configuration can be made in whichimages at all of the set frame rates are sent and the processing relatedto image synthesis and the like is omitted.

In the embodiments described above, the monitoring video also includes aframe-by-frame video such as a time-lapse video.

In the embodiments described above, the second exposure condition is setin cases in which there is a peak P of the brightness distribution thatis outside the predetermined range R. However, a configuration can bemade in which the second exposure condition is set in cases in whichthere are a plurality of peaks P of the brightness distribution.

In the embodiments described above, the first exposure condition is 18%gray. However, when the imaging target includes many dark locations ormany bright locations, the first exposure condition may be set asdesired by the user.

In the embodiments described above, the image processing device 10includes the imager 40. However, a configuration can be made in whichthe image processing device does not include an imager, and is connectedto an external imaging device via the communicator 50.

In the embodiments described above, the image processing device 10generates an image for the facial recognition device 80 that performstwo-dimensional facial recognition. However, a configuration can be madein which the image processing device 10 generates an image for a facialrecognition device that performs three-dimensional facial recognition.

In the embodiments described above, the additional frame is set in casesin which there is a peak P of the image V outside the predeterminedrange R. However, a configuration can be made in which the additionalframe is set in cases in which there are a plurality of peaks P of theimage V, in cases in which a peak of the image V is not found, and thelike.

In the embodiments described above, for the Positive exposure and theNegative exposure, the exposure values are adjusted by Equation (1).However, the exposure values may be changed in accordance with adifference between the value of the brightness level at the distributioncenter of the peak and the value at the center of the range ofbrightness level values. Alternatively, the Positive exposure andNegative exposure may be preset exposure values such as +3EV, −3EV, orthe like.

In the embodiments described above, in the image synthesis processings 1and 2, a face image trimmed from an image captured in an additionalexposure frame is combined with an image captured in a referenceexposure frame. However, a configuration can be made in which a targetsuch as a face image or the like that is captured under the referenceexposure is trimmed and combined with an image captured in an additionalexposure frame.

In the embodiments described above, the image synthesis processings 1and 2 and the like are performed by combining a face image trimmed froman image captured in an additional exposure frame with an image capturedin a reference exposure frame. However, a configuration can be made inwhich image synthesis is not performed, and the images are made tocorrespond by embedding coordinate information in a file tag of thetrimmed face image or the like.

In Embodiment 2 described above, when a face image is detected onlyunder the second exposure condition, the imaging under the firstexposure condition and the imaging under the second exposure conditionis alternately switched one frame at a time; and, when the same faceimage is detected under the first exposure condition and the secondexposure condition, and when the face image is not detected under thesecond exposure condition, switching is performed such that four framesare imaged under the first exposure condition and then one frame isimaged under the second exposure condition. However, the frequency ofthe switching of the frames can be set as desired. For example, aconfiguration can be made in which, when a face image is detected onlyunder the second exposure condition, the frames are switched such thatone frame is imaged under the first exposure condition and then fourframes are imaged under the second exposure condition, thereby placing agreater weighting on the second exposure condition than on the firstexposure condition. Alternatively, a configuration can be made in whichthe frequency at which the frames are switched is calculated inaccordance with face image detection frequency under each exposurecondition or the like.

In Embodiment 2 described above, when different face images are detectedunder the first exposure condition and the second exposure condition,and when a face image is detected only under the second exposurecondition, imaging under the first exposure condition and imaging underthe second exposure condition are alternately switched one frame at atime; and, when a face image is not detected under the second exposurecondition, switching may be carried out such that four frames are imagedunder the first exposure condition and then one frame is imaged underthe second exposure condition, or the frequency at which the frames areswitched may be changed by another condition. For example, when a faceimage under one exposure condition is detected at a predeterminedposition, switching may be carried out such that four frames are imagedunder that exposure condition and then one frame is imaged under anotherexposure condition; when a face image of a predetermined size or largeris detected under one exposure condition, switching may be carried outsuch that four frames are imaged under that exposure condition and thenone frame is imaged under another exposure condition; or the like.

In Embodiment 2 described above, the second exposure condition is set as+2EV. However, a configuration can be made in which a different value isset for the second exposure condition or the second exposure conditionis calculated by a method such as that in Embodiment 1. Additionally,two second exposure conditions may be set as in Embodiment 1, or threeor more second exposure conditions may be set. Furthermore, in theexposure correcting processing 2, fine adjustment of the frames may beperformed similar to step S11. Alternatively, a part or all of theprocessing of the exposure correction processing 2 may be incorporatedinto the exposure correction processing 1.

In Embodiment 2, an image captured under the first exposure condition isused for the background of the MAP image. However, a configuration canbe made in which an image captured under the second exposure conditionis used for the background. When the background is dark or bright, orthe like, there are cases in which it is preferable to use an imagecaptured under the second exposure condition for the background.

Note that the various functions of the image processing device 10 can beimplemented by a computer such as a typical personal computer (PC).Specifically, in the embodiments described above, an example isdescribed in which the programs of the exposure correction processings 1and 2 and the image processing, which are executed by the imageprocessing device 10, are stored in advance in the ROM of the storage30. However, a computer may be configured that is capable of realizingthese various features by storing and distributing the programs on anon-transitory computer-readable recording medium such as a compact discread-only memory (CD-ROM), a digital versatile disc (DVD), and amagneto-optical disc (MO), and reading out and installing these programson the computer.

The foregoing describes some example embodiments for explanatorypurposes. Although the foregoing discussion has presented specificembodiments, persons skilled in the art will recognize that changes maybe made in form and detail without departing from the broader spirit andscope of the invention. Accordingly, the specification and drawings areto be regarded in an illustrative rather than a restrictive sense. Thisdetailed description, therefore, is not to be taken in a limiting sense,and the scope of the invention is defined only by the included claims,along with the full range of equivalents to which such claims areentitled.

What is claimed is:
 1. An image processing device, comprising: a memoryconfigured to store a program; and at least one processor configured toexecute the program stored in the memory, wherein the processorconfigured to execute the program sets an exposure condition for imaginga monitoring video, acquires an image included in the monitoring videoimaged under the set exposure condition, and calculates a brightnessdistribution from the acquired image, the exposure condition includes(1) a first exposure condition for causing an imaging device to capturethe image using a first exposure value, and (2) a second exposurecondition for at least one of causing the imaging device to capture theimage using an exposure value higher than the first exposure value orcausing the imaging device to capture the image using an exposure valuelower than the first exposure value, and the processor sets the firstexposure condition or set the first exposure condition and the secondexposure condition as the exposure condition for imaging based on thebrightness distribution of the image captured under the first exposurecondition after the first exposure condition is set.
 2. The imageprocessing device according to claim 1, wherein the processor sets thesecond exposure condition when one or more peak of brightness levelvalue peaks, in the brightness distribution of the image captured underthe first exposure condition, is outside a predetermined range ofbrightness level values.
 3. The image processing device according toclaim 2, wherein the second exposure condition is an exposure conditionthat causes the image to be captured using the exposure value higherthan the first exposure value when one or more peak of the peaks is in arange lower than the predetermined range of brightness level values, andis an exposure condition that causes the image to be captured using theexposure value lower than the first exposure value when one or more peakof the peaks is in a range higher than the predetermined range ofbrightness level values.
 4. The image processing device according toclaim 2, wherein the processor changes the exposure value of the secondexposure condition in accordance with a difference between a brightnesslevel value of a distribution center of the peaks and a value of acenter of the predetermined range of brightness level values.
 5. Theimage processing device according to claim 1, wherein the processor,when the brightness distribution of the image captured under the firstexposure condition includes a plurality of brightness level value peaks,sets the second exposure condition based on a brightness level valuepeak.
 6. The image processing device according to claim 1, wherein theprocessor reduces a frequency of imaging under the second exposurecondition or increases a frequency of imaging under the first exposurecondition when a subject is not included in the image captured under thesecond exposure condition or a same subject is included in both theimage captured under the first exposure condition and the image capturedunder the second exposure condition.
 7. The image processing deviceaccording to claim 1, wherein the processor increases a frequency ofimaging under the second exposure condition or reduces a frequency ofimaging under the first exposure condition when a subject is includedonly in the image captured under the second exposure condition.
 8. Theimage processing device according to claim 1, wherein the processorcombines an image captured under the second exposure condition with theimage captured under the first exposure condition.
 9. The imageprocessing device according to claim 6, wherein the processor combines(i) a target that is not detected in an image captured under oneexposure condition of the first and second exposure conditions yet isdetected in an image captured under another exposure condition of thefirst second exposure conditions with (ii) an image captured under theone exposure condition.
 10. The image processing device according toclaim 9, wherein the target is the subject.
 11. The image processingdevice according to claim 8, wherein the processor combines (i) a firstimage that is not detected in an image captured under one exposurecondition of the first and second exposure conditions yet is detected inan image captured under another exposure condition of the first andsecond exposure conditions and (ii) a second image that is detected inthe image captured under the first exposure condition and is detected inthe image captured under the second exposure condition with (iii) theimage captured under the one exposure condition, a value of a peak of abrightness level of the second image in the image captured under theother exposure condition being closer to a median value of brightnesslevels under each of the exposure conditions than a value of a peak of abrightness level of the second image in the image captured under the oneexposure condition.
 12. An image processing method, comprising: settingan exposure condition; acquiring an image captured under the setexposure condition; and calculating a brightness distribution from theacquired image, wherein the exposure condition includes (1) a firstexposure condition for causing an imaging device to capture the imageusing a first exposure value, and (2) a second exposure condition for atleast one of causing the imaging device to capture the image using anexposure value higher than the first exposure value or causing theimaging device to capture the image using an exposure value lower thanthe first exposure value, and the first exposure condition is set or thefirst exposure condition and the second exposure condition are set asthe exposure condition for imaging, based on the brightness distributionof the image captured under the first exposure condition after the firstexposure condition is set.
 13. A non-transitory computer-readablerecording medium having stored therein a program that causes a processorof a computer to function so as to: set an exposure condition; acquirean image captured under the set exposure condition; and calculate abrightness distribution from the acquired image, wherein the exposurecondition includes (1) a first exposure condition for causing an imagingdevice to capture the image using a first exposure value, and (2) asecond exposure condition for at least one of causing the imaging deviceto capture the image using an exposure value higher than the firstexposure value or causing the imaging device to capture the image usingan exposure value lower than the first exposure value, and the firstexposure condition is set or the first exposure condition and the secondexposure condition are set as the exposure condition for imaging, basedon the brightness distribution of the image captured under the firstexposure condition after the first exposure condition is set.
 14. Animage processing device, comprising: a memory configured to store aprogram; and at least one processor configured to execute the programstored in the memory, wherein the processor configured to execute theprogram sets an exposure condition, acquires an image captured under theset exposure condition, and calculates a brightness distribution fromthe acquired image, the exposure condition includes (1) a first exposurecondition for causing an imaging device to capture the image using afirst exposure value, and (2) a second exposure condition for at leastone of causing the imaging device to capture the image using an exposurevalue higher than the first exposure value or causing the imaging deviceto capture the image using an exposure value lower than the firstexposure value, and the processor sets one or more of the first exposurecondition and the second exposure condition included in the exposurecondition by setting the first exposure condition and then, in additionto the first exposure condition, setting the second exposure conditionbased on the brightness distribution of the image captured under thefirst exposure condition.